Boosting Generic Visual-Linguistic Representation with Dynamic Contexts

Pretraining large models on generous multi-modal corpora has accelerated the development of visual-linguistic (VL) representation and achieved great success various vision-and-language downstream tasks. Learning these is usually executed by predicting randomly masked words captions or patches in images. Such approaches, nevertheless, seldom explore supervision causalities behind caption descriptions procedure generating events beyond still In this work, we endow pretrained with high-level cognition delving into dynamic contexts to model visual linguistic uniformly. Specifically, format dynamic contexts an image as sentences describing xmlns:xlink="http://www.w3.org/1999/xlink">before , xmlns:xlink="http://www.w3.org/1999/xlink">on xmlns:xlink="http://www.w3.org/1999/xlink">after image. Unlike traditional caption-wise similarity, propose a novel contexts-based similarity (DCS) metric, which correlation potential causes effects besides immediate content are considered measure relevance among DCS can be further simplified parameterizing event continuity relax requirements dense contextual annotations. A new pre-task designed minimize feature distances dynamically relevant images incorporate causality commonsense knowledge VL learning. Models based our significantly outperform typical multiple cross-modal tasks, including conventional reasoning (VCR), question answering (VQA), zero-shot image-text retrieval, extended / ordering